FAOSTAT 2.3
A revitalisation of the API wrapper of the FAOSTAT API
The FAOSTAT package is an important part of the Food and Agriculture Organization (FAO)’s image that is being maintained, but requires a makeover. Here an updated version, 2.3.0, of the package is presented with repaired access to FAO’s API, new functions and preparation for a complete overhaul in 3.0.0. The package has been modernised according to new coding conventions with improved dependencies, documentation and tests. Old useless functions have been pruned and the package is now firmly focused on providing an interface to FAO data to users of R.
Other formats
Project background
The motivation for this project came from a Data Mining project from UniLaSalle. It was suggested that we use FAO1 data from their statistical platform FAOstat2. As R was the language of choice, the obvious port of call was the FAOSTAT package3 (Kao, Gheri, and Gesmann 2022), developed by employees at FAO.
1 Food and Agriculture Organization of the United Nations
2 Food and Agriculture Organization Corporate Statistical Database
3 For the purposes of clarity, this document will use the style “FAOSTAT” for the R package and “FAOstat” for the statistical platform
However, the FAOSTAT package did not work. It could not download data from the API and could only download bulk data with the entirety of a dataset in one go. For the particular dataset we were interested in, we found that there was a discrepancy between the data in the bulk download and the data on the web platform.4
4 This discrepancy has been fixed as of 2023-03-10
Eventually it became necessary to use the same API that the FAOstat website uses to pull data. This method worked and it became clear that it could be used to revitalise the FAOSTAT package and part of an effort to restore it to full functionality.
FAOstat
FAOstat is FAO’s web-based statistical platform for the free dissemination of food and agriculture statistics. This data is obtained from questionnaires that FAO distributes throughout the world every year (Food and Agriculture Organization of the United Nations 2019). Some of its data also comes from imputations and models where data is not available, but official country data takes precedence.
The FAOstat service is a public-facing aspect of FAO, with an overall trend of increasing citations in academic papers year on year with 21400 citations by 2021 (Figure 1).
This platform uses a REST API internally to communicate with its database as well as providing a set of zip files with the entirety of certain datasets in order to reduce the load on the database. This REST API allows the website to generate CSVs as well as to allow exploration of the data via interactive graphs (Figure 2).
FAOSTAT package
The FAOSTAT package is an API wrapper to pull data from FAOSTAT into a R session. It can also perform small necessary tasks such as country code conversion and coalescing data from different country groups.5
5 For example, China may be just the mainland or may include Taiwan (Chinese Taipei), Hong Kong and Macao
History
The FAOSTAT package was originally developed in 2013 as a tool to source data for the SYB6 project. The yearbooks are yearly summaries of the worldwide state of agriculture for that year. At the time, they were manually typeset and compiled. The new SYB project was to use a combination of LaTeX, knitr and R to automatically pull data from FAOSTAT and other data sources such as the World Bank. This data would be then be transformed and processed to create graphs and tables before finally formatting and typesetting to create a finished product which could then be printed.7. Given that this use case no longer exists, the primary use of this package is for researchers and other R users to read data from FAOstat in a clean way that makes it easier to move to analysis afterwards.
6 Statistical Year Book
7 The author has no insight into the current production of the SYB, but they are still being produced and can be found on the FAO website
It is a reasonably popular package in the 86th percentile of all packages on CRAN on 2023-04-01 by downloads. In total, the package has been downloaded over 50 000 times with a peak 121 daily downloads on 2019-05-15. (Li 2023)
The package was maintained by Michael Kao, the author, from 2013 to 2014. In 2014, it was maintained by Filippo Gheri before passing to Paul Rougieux (the current maintainer) in 2020.
While it was originally hosted on Github under Michael Kao’s personal account, It is currently hosted on GitLab under Paul Rougieux’s personal account.
Current state
The FAOSTAT package has only a shadow of its former functionality. While it has retained the ability to download and process zip files and country code processing functions,8 its capacities are limited by the following issues:
8 For a full description of the status of individual issues, please see the GitLab issue #20 Remove functions linked to defunct uses of FAOSTAT
Functionality locked to the Statistical Yearbook
A number of functions are simply designed to pull in data from other sources such as the World Bank and to process that data into a format easily consumed by the Statistical Yearbook. As the yearbook no longer uses the FAOSTAT package, these functions have no further purpose, serving only to clog up the package and its help files.
Functionality powered by local files
Many uses of FAOSTAT require data outside of the data that comes directly from FAOSTAT. The major use case is for code conversions. There are two main code types that require conversion:
- Country codes
- FAO: FAO’s internal codes for countries9
- M49: The UN standard country codes
- ISO2 & ISO3: 2 and 3 letter country codes
- Item codes
- FAO: Internal FAO codes to describe commodities
- CPC: Central Product Classification code (United Nations Statistical Division 2005)
9 For further details about FAO and how it handles country identification, see FAO’s NOCS database
The conversions are not dynamically taken from the API but rather stored in a fixed file. This makes them vulnerable to code or name changes in the future such as the name change of Swaziland to Eswatini in 2018 (United Nations 2018).
Change of FAOstat API
The FAOSTAT package is currently configured to access a now-defunct API10 (FAOSTAT3). As a result, it has no methods of retrieving data from the FAOstat platform with the sole exception of the bulk zip downloads which have been since adapted to use the current platform.
10 Originally hosted at faostat3.fao.org
Other issues
The FAOSTAT package is currently maintained by Paul Rougieux who has done an excellent job of keeping the package afloat. However, it’s a small project done in his spare time, so he hasn’t been able to make the time to do a full overhaul. In addition, he isn’t an employee of FAO, but rather of the European Commission. As a face that FAO shows to the world, it seems reasonable that it be placed under its guidance.
As the package is old, it also has a number of dependencies that are unused or will be after redundant functions are removed. Other dependencies are simply no longer developed or have been superseded by newer packages.
Project goals
There are four main goals of this project:
- Fix up core functions
- Triage existing functions
- Characterise relevant aspects of the new API to wrap
- Transfer maintainership
Fix up core functions
The first priority to bring the FAOSTAT package to an acceptable level of functionality is to repair the most basic and most used functions.11
11 The functions below have been paired with a corresponding Gitlab issue that gives more context
Functions should not only be repaired but also renamed to have a consistent naming system.
Triage existing functions
As described in the Current state section, many functions no longer work and need to be removed. This involves a thorough listing of them and their functionality and assessment of their usefulness.
Characterise relevant aspects of the new API to wrap
Full overhauling the package is not in scope for FAOSTAT 2.3.0, but rather reserved for the next major version FAOSTAT 3.0.0. 15 This requires the additional step of inspecting the API manually as it doesn’t have complete documentation.
15 For a full set of changes expected for FAOSTAT 3.0.0, see the GitLab milestone page
Transfer maintainership
Uploading a package on CRAN16 requires the permission of the maintainer. At the upload step, an email is sent to the maintainer of the last recorded package version to confirm that the upload was authorised. Only then can the new version proceed to the registration step.
16 Comprehensive R Archive Network, the centralised repository where the majority of R users download R packages
Methods
Talked with existing maintainer Interviewed package author
Please see issues in milestone: https://gitlab.com/paulrougieux/faostatpackage/-/issues/?sort=created_date&state=closed&milestone_title=2.3.0&first_page_size=20
Examining the API
Refactor essential functions
- getFAO -> read_fao
- FAOsearch -> search_fao
Had to repair zip function because of fao search
Add caching
One of the drawbacks of moving from internal data to pulling from the API is the need to read in data every time a lookup table or other piece of reference data is required. This slows down scripts as it’s forced to wait for a reply from the FAOstat server. To mitigate that, caching was implemented. 17 Now instead of pulling every time, the server is polled only once and the response is kept and referred to for subsequent queries.
- Add column metadata
Documented using roxygen2 (Wickham et al. 2022), tests written using testthat (Wickham 2011)
- examples
Use case
- Check health of zip data
- Display availability of certain country data (shiny app)
- Graph per country
- Colour by flag
Future work
- Release 3.0.0
- Fully integrating with the new API
- Publishing a paper in JOSS
- Publish news in rweekly
- Publish FAO news alert
- Move package to FAO repository
Funding declaration
This project has been funded by the Food and Agriculture Organization of the United Nations and the author is grateful for their help in reviving it.